Non-transitory computer-readable recording medium, performance display method, and performance display apparatus

ABSTRACT

A video reproduction apparatus determines a position of a joint of a performer based on a sensing result by a sensor. The video reproduction apparatus calculates an angle formed by a joint used for scoring an element of an athletic event by using calculation information regarding a calculation formula for calculating the angle and the determined position of the joint of the performer. The video reproduction apparatus displays a performance image of the performer acquired from a camera and a scoring image having the angle displayed on a 3D model image of the performer which is generated according to the sensing result so as to be compared with each other.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of InternationalApplication PCT/JP2017/018259, filed on May 15, 2017, and designatingthe U.S., the entire contents of which are incorporated herein byreference.

FIELD

The present invention relates to a performance display program, aperformance display method, and a performance display apparatus.

BACKGROUND

Conventionally, in sports or the like where a player performs a sequenceof elements and a judge gives a score, the success or the degree ofperfection of a highly difficult element largely affects a high score ora high rank. For example, six events are formed for men, four events forwomen, and in events other than vault for men and women, one performanceis formed of more than ten elements performed in succession. In ascoring rule, criteria are defined for the type, the group, the number,and the execution of elements, and a criterion is set for each elementto determine a score. A judge gives a score to a performance in a shorttime after the player finishes the performance.

Nowadays, there is a known technology of acquiring information about theattitude, the movement, or the like, of a person by using 3D sensing toassist a sport player, and the like. For example, it is used to checkthe degree of perfection of an element as it acquires the movement andthe like of a gymnast when he performs the element during practice.

Patent Literature 1: Japanese Laid-open Patent Publication No.2010-264088

SUMMARY

According to an aspect of an embodiment, a non-transitorycomputer-readable recording medium stores therein a performance displayprogram that causes a computer to execute a process. The processincludes determining a position of a joint of a performer based on asensing result by a sensor; calculating an angle formed by a joint usedfor scoring an element of an athletic event by using calculationinformation regarding a calculation formula for calculating the angleand the determined position of the joint of the performer; anddisplaying a performance image of the performer acquired from a cameraand a scoring image having the angle displayed on a 3D model image ofthe performer which is generated according to the sensing result so asto be compared with each other.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram that illustrates a video reproduction apparatusaccording to a first embodiment.

FIG. 2 is a functional block diagram that illustrates a functionalconfiguration of the video reproduction apparatus according to the firstembodiment.

FIG. 3 is a diagram that illustrates an example of the informationstored in a joint definition data DB.

FIG. 4 is a diagram that illustrates an example of the informationstored in a joint position data DB.

FIG. 5 is a diagram that illustrates an example of the informationstored in a video data DB.

FIG. 6 is a diagram that illustrates an example of the informationstored in a scoring rule DB.

FIG. 7 is a diagram that illustrates an example of the informationstored in a calculation formula DB.

FIG. 8 is a diagram that illustrates an example of calculation of ajoint angle in a final phase (front-side support) of pommel horse.

FIG. 9 is a diagram that illustrates an example of calculation of ajoint angle during forward upward circling to support at ring height onstill rings.

FIG. 10 is a diagram that illustrates a display example of an outputscreen.

FIG. 11 is a diagram that illustrates a display example of an enlargeddisplay.

FIG. 12 is a diagram that illustrates a display example of pointdeduction.

FIG. 13 is a flowchart that illustrates the flow of preprocessing.

FIG. 14 is a flowchart that illustrates the flow of a process toregister a joint position.

FIG. 15 is a flowchart that illustrates the flow of a comparison displayprocess.

FIG. 16 is a flowchart that illustrates the flow of a process to displayan angle formed by a joint.

FIG. 17 is a diagram that illustrates an example of a hardwareconfiguration.

DESCRIPTION OF EMBODIMENTS

It is, however, difficult for even an expert such as a judge toaccurately check the joint angle of a player and give a score to theperformance in accordance with the scoring rule that is defined for eachevent or element, and therefore scores vary depending on individualjudges.

For example, the body of a gymnast makes a complicated movement in threedimensions, and each gymnast has a different physical size and movementfor an element. Therefore, it is not easy for even a judge who islicensed as a judge to give a score after determining, in a short time,that a certain element has been accurately performed in accordance witha scoring rule. For this reason, scores for individual elementssometimes vary depending on a judge and, in some cases, a playerexpresses an opposition (inquiry) to the judge in a predetermined methodso as to make a detailed rejudgement on the score.

According to an aspect, embodiments provide a performance displayprogram, a performance display method, and a performance displayapparatus capable of providing information for conducting scoring in theuniform manner in accordance with a scoring rule.

With reference to the drawings, an embodiment of a performance displayprogram, a performance display method, and a performance displayapparatus according to the present invention is described below indetail. Furthermore, the present invention is not limited to theembodiment. Moreover, each embodiment may be combined as appropriate aslong as the consistency is ensured.

First Embodiment

[Description of Video Reproduction Device]

FIG. 1 is a diagram that illustrates a video reproduction apparatus 10according to a first embodiment. The video reproduction apparatus 10illustrated in FIG. 1 is an example of a computer, such as a server, atablet terminal, or a smartphone, and it is an example of a performancedisplay apparatus. The video reproduction apparatus 10 uses a 3D (ThreeDimensions) laser sensor to determine the movement or the position of ajoint of a player in three dimensions and in chronological order andstores it as joint position data in a joint table in advance.Furthermore, the video reproduction apparatus 10 obtains, as a rule, thescoring rule for each athletic event and registers it in a database soas to generate, as indexes, the type of element, the joint and the angleused for scoring an element, a point deduction due to angle deviation,or the like.

The above video reproduction apparatus 10 is used for the judge to viewthe video, check the score, and make a rejudgement in the case of anopposition to the score announced after the performance ends.Specifically, the video reproduction apparatus 10 stores calculationinformation regarding a calculation formula for calculating the angleformed by a joint used for scoring an element in an athletic event.Then, the video reproduction apparatus determines the position of ajoint of the performer from the performance image that is a specificframe designated in the performance video of the performer captured by avideo camera. Then, the video reproduction apparatus 10 calculates theangle formed by a joint by using the determined position of the joint ofthe performer and the calculation information and displays theperformance image designated in the performance video and the scoringimage having the angle displayed on the performance image so as to becompared with each other.

Specifically, the video reproduction apparatus 10 not only simplyvisualizes the movement and the position of each joint during theperformance while determining the attitude and the movement of theperformer due to 3D sensing with the 3D laser sensor but also presents,to a judge, the information that aids in making a determination forscoring the execution of the element in accordance with a scoring rule.As a result, the video reproduction apparatus 10 is capable of providingthe information for conducting scoring in the uniform manner inaccordance with the scoring rule.

[Functional Configuration]

FIG. 2 is a functional block diagram that illustrates a functionalconfiguration of the video reproduction apparatus 10 according to thefirst embodiment. As illustrated in FIG. 2, the video reproductionapparatus 10 includes a communication unit 11, a storage unit 12, and acontrol unit 20.

The communication unit 11 is a communication interface, or the like,which controls communications with other devices. For example, thecommunication unit 11 receives a performance video captured by a videocamera and receives a performance video captured by a video camera.Furthermore, the communication unit 11 receives joint position data thatis a result of 3D sensing with a 3D laser sensor. Moreover, thecommunication unit 11 receives a scoring rule from the administrator'sterminal, or the like.

The storage unit 12 is an example of a storage device that stores aprogram and data, and it is, for example, a memory or a hard disk. Thestorage unit 12 stores a joint definition data DB 13, a joint positiondata DB 14, a video data DB 15, a scoring rule DB 16, and a calculationformula DB 17. Furthermore, the calculation formula DB 17 is an exampleof a scoring storage unit.

The joint definition data DB 13 is a database that stores definitioninformation for determining each joint in a skeleton model. Thedefinition information stored here may be measured for each performerduring 3D sensing with a 3D laser sensor or may be defined by using askeleton model of the typical body.

FIG. 3 is a diagram that illustrates an example of the informationstored in the joint definition data DB 13. As illustrated in FIG. 3, thejoint definition data DB 13 stores the information on numbering of eachjoint determined by using a known skeleton model. For example, asillustrated in FIG. 3, number 7 is assigned to the right shoulder joint(SHOULDER_RIGHT), number 5 is assigned to the left elbow joint(ELBOW_LEFT), number 11 is assigned to the left knee joint (KNEE_LEFT),and number 14 is assigned to the right hip joint (HIP_RIGHT). Here,according to the embodiment, the X-coordinate of the number-7 rightshoulder joint is sometimes described as X7, the Y-coordinate as Y7, andthe Z coordinate as Z7. Furthermore, numbers in a dotted line are of ajoint, or the like, which are not used for scoring although they aredetermined from the skeleton model.

The joint position data DB 14 is a database that stores the positiondata on each joint acquired from video data on the performer andtemporal changes in the joint position data. FIG. 4 is a diagram thatillustrates an example of the information stored in the joint positiondata DB 14. As illustrated in FIG. 4, the joint position data DB 14stores “record ID, time, X0, Y0, Z0, . . . , X17, Y17, Z17” in a relatedmanner. “Record ID” stored here is the information for identifying videodata, and “time” is the time when the positional information on a jointis determined. “X0, Y0, 20, . . . , X17, Y17, 217” are the XYZcoordinates of each joint, and “X0, Y0, 20” are the coordinates of thejoint with number 0 illustrated in FIG. 3.

FIG. 4 illustrates temporal changes in each joint in the video data withnumber 001, and it indicates that, at the time “00:00:00”, the positionsof the respective joints are “X0=100, Y0=20, Z0=0, . . . , X17=200,Y17=40, Z17=5”. Furthermore, it indicates that, at the time “00:00:33”,the positions of the respective joints have moved to “X0=101, Y0=25,Z0=5, . . . , X17=202, Y17=39, Z17=15”.

The video data DB 15 is a database that stores video data on aperformer. FIG. 5 is a diagram that illustrates an example of theinformation stored in the video data DB 15. As illustrated in FIG. 5,the video data DB 15 stores “record ID, event, player ID, time, videofile” in a related manner. “Record ID” stored here is the informationfor identifying video data, “event” indicates the event of a performercaptured by the video camera, “player ID” is the information foridentifying a performer, “time” is the information indicating aperformance time, and “video file” is the information for determiningvideo data. The case of FIG. 5 indicates that the video file “001_pommelhorse_1122.mp4” with the record ID “001” is the video data on “pommelhorse” performed by the player with the player ID “1122” and it is theperformance data in “10 minutes 99 seconds”.

The scoring rule DB 16 is a database that stores a scoring rule for eachevent. FIG. 6 is a diagram that illustrates an example of theinformation stored in the scoring rule DB 16. As illustrated in FIG. 6,the scoring rule DB 16 stores “scoring rule ID, event, action, scoringtarget, threshold (minor fault, intermediate fault, major fault), pointdeduction (minor fault, intermediate fault, major fault)” in a relatedmanner.

“Scoring rule ID” stored here is the information for identifying ascoring rule. “Event” is the information for determining the targetevent to be scored. “Action” is the information for determining thetarget action to be scored. “Scoring target” is the information fordetermining the element name, the score item, or the like, for which theangle, which is the target to be scored, is calculated. “Threshold(minor fault, intermediate fault, major fault)” is the informationindicating a threshold for the angle that is the target for pointdeduction. “Point deduction (minor fault, intermediate fault, majorfault)” is the information indicating the details of point deduction.

In the case of FIG. 6, the scoring rule ID “001001” indicates that“angle deviation” in “final phase” of “longitudinal circling” of “pommelhorse” is the scoring target. This example indicates that a judgement ismade such that point deduction is “0.1” in the range of the angle from15° (15 degrees) to 30°, point deduction is “0.3” in the range of theangle from 31° to 45°, and point deduction is “0.5” when the angle ismore than 45°.

The calculation formula DB 17 is a database that stores a calculationformula for the angle between joints, which is the target to be scored,i.e., the angle formed by joints. FIG. 7 is a diagram that illustratesan example of the information stored in the calculation formula DB 17.As illustrated in FIG. 7, the calculation formula DB 17 stores “scoringrule ID, calculation formula, reference coordinates, display plane,number of times of measurements” in a related manner. “Scoring rule ID”stored here is the information for identifying a scoring rule.“Calculation formula” is a calculation formula for the target angle tobe scored. “Reference coordinates” are coordinates that serve as thereference for displaying the angle. “Display plane” is a plane fordisplaying the calculated angle. “Number of times of measurements” isthe number of times of calculations per scoring.

In the example of FIG. 7, the scoring rule ID “001001” indicates thatthe reference coordinates are of the joint with number 2 and the valueof “flag” dynamically changes in cases where the direction of circlingis in a clockwise direction (CW) and in a counterclockwise direction(CCW). Furthermore, it indicates that, for the display, it is displayedon the XY plane and the number of times of measurements is one.

Furthermore, the scoring rule ID “002001” indicates that the referencecoordinates are of the joint with number 9 and the difference betweenthe largest value and the smallest value (static attitude) of θ_(R) isthe scoring target. Furthermore, it indicates that the number of timesof measurements is twice as the largest value and the smallest value(static attitude) of θ_(R) are calculated and, for the display, it isdisplayed on the YZ plane.

The control unit 20 is a processing unit that controls the overall videoreproduction apparatus 10, and it is for example a processor. Thecontrol unit 20 includes a preprocessing unit 21, a relating unit 22, acomparison display unit 23, and an angle calculating unit 24.Furthermore, the preprocessing unit 21, the relating unit 22, thecomparison display unit 23, and the angle calculating unit 24 areexamples of an electronic circuit such as a processor or examples of aprocess executed by a processor. Moreover, the angle calculating unit 24is an example of a determining unit and a calculating unit, and thecomparison display unit 23 is an example of a display control unit.

The preprocessing unit 21 is a processing unit that previously acquiresdefinition information on a joint. Specifically, the preprocessing unit21 measures a joint of the performer during 3D sensing with the 3D lasersensor. Then, the preprocessing unit 21 numbers the joint of theperformer in accordance with the typical skeleton model and stores it inthe joint position data DB 14.

Furthermore, the preprocessing unit 21 acquires the performance video ofthe performer captured by using the video camera and stores it as videodata in the storage unit 12. Furthermore, the preprocessing unit 21executes 3D sensing with the 3D laser sensor on the performance capturedby using the video camera to acquire temporal changes in the jointposition data on the performer and stores it in the storage unit 12.

The relating unit 22 is a processing unit that relates the performancevideo with the joint position data during the performance. For example,the relating unit 22 assigns the record ID for relating the video dataacquired by the preprocessing unit 21 and the joint position data on theperformer acquired from the video data. Then, the relating unit 22relates the record ID with the information for determining the videodata and stores them in the video data DB 15 and relates the record IDwith the joint position data and stores them in the joint position dataDB 14.

The comparison display unit 23 is a processing unit that displays thevideo data on the performance and the joint position data in the videodata so as to be compared with each other. Specifically, the comparisondisplay unit 23 divides the video data into each frame (performanceimage) and displays it. Furthermore, the comparison display unit 23acquires the joint position data corresponding to each frame from thejoint position data DB 14, applies it to the skeleton model to generatea 3D model, and displays the generated 3D model. Specifically, thecomparison display unit 23 displays the specific frame selected by thejudge or the like from the frames forming the video data alongside ofthe 3D model of the joint position data on the performer at that point.

The angle calculating unit 24 is a processing unit that calculates theangle, which is the target to be scored, with respect to the 3D model ofthe joint position data on the performer corresponding to the selectedframe and displays it on the 3D model. Specifically, the anglecalculating unit 24 determines the event in the selected frame from thevideo data DB 15 and determines the scoring rule corresponding to thedetermined event from the scoring rule DB 16. Here, when multiplescoring rules are determined, the angle calculating unit 24 prompts thejudge or the like to select the scoring rule to be applied. Then, theangle calculating unit 24 determines the calculation formulacorresponding to the selected scoring rule from the calculation formulaDB 17 and, by using the determined calculation formula, calculates thetarget angle to be scored.

For example, in a case where the scoring rule “001001” is selected, theangle calculating unit 24 calculates the angle by using the calculationformula in the first line of FIG. 7. The angle calculated by thecalculation formula is the angle obtained from the positions of thechest and the waist in the “final phase” of the longitudinal circling ofpommel horse, and it is calculated based on the coordinates with thejoint number 2 and the coordinates with the joint number 0. Then, theangle calculating unit 24 causes the calculated angle to be displayed onthe XY plane of the 3D model.

FIG. 8 is a diagram that illustrates an example of the calculation ofthe joint angle in the final phase (front-side support) of pommel horse.In the example illustrated in FIG. 8, the longitudinal support is abasis. As illustrated in FIG. 8(a), the angle of the target joint to bescored in the final phase is an angle θ formed between the straight lineconnecting the chest (the joint number 2) and the waist (the jointnumber 0) and the horizontal direction of a base A of the pommel horse.The angle θ is calculated by the calculation formula in the first lineof FIG. 7. Furthermore, it is determined that there is no fault and apoint deduction is 0 when the angle θ is less than 15° (FIG. 8(b)), itis determined that there is a minor fault and a point deduction is 0.1points when the angle θ is between 15° and 300 (FIG. 8(c)), it isdetermined that there is an intermediate fault and a point deduction is0.3 points when the angle θ is between 31° and 45° (FIG. 8(d)), and itis determined that there is a major fault and a point deduction is 0.5points when the angle θ is more than 45° (FIG. 8(e)).

Another example is described; the angle calculating unit 24 calculatesthe angle by using the calculation formula in the second line of FIG. 7in a case where the scoring rule “002001” is selected. The anglecalculated by this calculation formula is the difference between thelargest angle and the smallest angle (static attitude) of the “shoulderposition” from the swing on still rings to the static attitude, and itis calculated based on the coordinates of the joint number 9 and thecoordinates of the joint number 7. Then, the angle calculating unit 24causes the calculated angle to be displayed on the YZ plane of the 3Dmodel.

FIG. 9 is a diagram that illustrates an example of the calculation ofthe joint angle during forward upward circling to support at ring heighton still rings. As illustrated in FIG. 9, the angle θ_(R) of the targetjoint to be scored during forward upward circling to support at ringheight on still rings is the angle formed between the straight lineconnecting the right shoulder (the joint number 9) and the right hand(the joint number 7) and the direction (Y direction) horizontal to theground. The angle θ_(R) is calculated by the calculation formula in thesecond line of FIG. 7. As illustrated in the second line of FIG. 7, thenumber of times of calculations for the angle θ is twice. This isbecause the difference between the largest angle and the smallest angleis the target to be scored.

Specifically, the angle calculating unit 24 calculates the largest angleθ_(R) from the frame that is designated by the judge as the target fordetermining the largest angle and calculates the angle θ_(R) at thestationary time from the frame that is designated by the judge as thetarget for determining the static attitude. Then, the difference is thetarget to be scored. For example, it is determined that there is nofault and a point deduction is 0 when the difference between the largestangle θ_(R) and the angle θ_(R) at the stationary time is less than 15°,it is determined that there is a minor fault and a point deduction is0.1 points when the difference is between 15° and 30°, it is determinedthat there is an intermediate fault and a point deduction is 0.3 pointswhen the difference is between 31° and 45°, and it is determined thatthere is a major fault and a point deduction is 0.5 points when thedifference is more than 45°.

[Screen Display Example]

Next, an example of the screen display presented on the display, or thelike, by the comparison display unit 23 is described. FIG. 10 is adiagram that illustrates a display example of the output screen. Thecomparison display unit 23 generates and displays a comparison displayscreen B illustrated in FIG. 10. The comparison display screen Bincludes information such as the name of a performer, the team hebelongs to, or the event performed. The comparison display screen Bincludes a video area C for displaying the image of a selected frame, aselection area D for displaying a scoring rule in a selectable manner,an area P for displaying each frame, and a scroll bar Q for selecting aframe. Furthermore, the XYZ plane is previously determined on thescreen. For example, in the case of pommel horse, it is determined thatthe horizontal direction of the pommel horse is the X axis, the verticaldirection is the Y axis, and the depth direction is the Z axis.

Specifically, the comparison display unit 23 displays the video (image)corresponding to the frame selected by the judge on the video area C ofthe comparison display screen B. Furthermore, the comparison displayunit 23 displays the image of each frame forming the video data on thearea P of the comparison display screen B. Moreover, the comparisondisplay unit 23 displays, on the area C, the image of the frame selectedfrom the frames displayed on the area P by a user, such as a judge,moving the scroll bar 0.

Furthermore, the comparison display unit 23 displays the skeleton modelcorresponding to the image of the frame displayed on the video area Calongside of the image in the video area C. Here, the comparison displayunit 23 displays images of the skeleton model viewed from multipleviewpoints that are previously determined. Furthermore, the anglecalculating unit 24 uses the calculation formula for the scoring ruleselected from the scoring rules displayed on the selection area D todisplay the angle on each image of the skeleton model displayed. Thatis, the judge selects the scoring rule displayed on the selection area Dso as to display the desired angle.

For example, when the judge selects the frame in 08 seconds and ascoring rule D1, the comparison display unit 23 displays the image ofthe frame in 08 seconds on the area C and displays the skeleton modelfor the image. Then, the angle calculating unit 24 acquires the jointposition data corresponding to the frame from the joint position data DB14, calculates the target angle to be scored by using the calculationformula corresponding to the scoring rule D1, and then displays it bybeing superimposed on the displayed image of the skeleton model.

Then, when the judge selects the frame in 09 seconds while maintainingthe scoring rule D1, the comparison display unit 23 displays the imageof the frame in 09 seconds on the area C and displays the skeleton modelfor the image. Then, the angle calculating unit 24 displays the anglecalculated by using the joint position data corresponding to the frameby being superimposed on the displayed image of the skeleton model.

Furthermore, when the judge selects a scoring rule D2 while selectingthe frame in 09 seconds, the comparison display unit 23 maintains thedisplay, and the angle calculating unit 24 acquires the joint positiondata corresponding to the frame from the joint position data DB 14,calculates the target angle to be scored by using the calculationformula corresponding to the scoring rule D2, and then displays it bybeing superimposed on the displayed image of the skeleton model.

Furthermore, in a case where a single frame is selected, the anglecalculating unit 24 may previously calculate the angle with regard tothe previous and the next frames. For example, when the frame in 08seconds is selected, the angle calculating unit 24 may also calculatethe angle using the skeleton model of the frame in 07 seconds and theangle using the skeleton model of the frame in 09 seconds in addition tothe angle using the skeleton model of the frame in 08 seconds.

Thus, the previous calculation allows a decrease in the time tocalculate the angle due to frame switching and a reduction in theworkload of the judge. Furthermore, even in a case where the judgechecks angles by frequently switching frames, the checking task may beperformed without stress.

Furthermore, the comparison display unit 23 may also display, in anenlarged manner, the image of the skeleton model selected by doubleclick or the like from multiple images of the skeleton model. FIG. 11 isa diagram that illustrates a display example of the enlarged display. Asillustrated in FIG. 11, in a case where the image of the skeleton modelon the top is selected in the state of FIG. 10, the comparison displayunit 23 displays the image of the skeleton model on the top in anenlarged manner. Furthermore, the selection of a frame may be executedby pressing the button displayed under the area B.

Furthermore, the angle calculating unit 24 may also change the displayformat of the angle in accordance with the degree of point deduction.FIG. 12 is a diagram that illustrates a display example of pointdeduction. For example, the angle calculating unit 24 displays the anglein blue in the case of a minor fault (FIG. 12(a)), displays the angle inyellow in the case of an intermediate fault (FIG. 12(b)), and displaysthe angle in red in the case of a major fault (FIG. 12(c)). This makesit possible to visually display the degree of point deduction and toreduce the workload of the judge. Furthermore, the colors may beoptionally changed.

[Flow of Preprocessing]

FIG. 13 is a flowchart that illustrates the flow of preprocessing. Asillustrated in FIG. 13, the preprocessing unit 21 receives aninstruction to start to capture a performance (S101) and then assigns arecord ID (S102). Then, the preprocessing unit 21 acquires jointposition data with the 3D laser sensor (S103) while acquiring video datawith the video camera (S104).

Then, the relating unit 22 relates the joint position data with thevideo data and stores them in the joint position data DB 14 and thevideo data DB 15, respectively (S105). Furthermore, the timing in whichthe record ID is assigned may be optionally changed.

[Flow of Process to Register Joint Position]

FIG. 14 is a flowchart that illustrates the flow of a process toregister a joint position. As illustrated in FIG. 14, the preprocessingunit 21 assigns the scoring rule ID (S201), receives, from theadministrator or the like, points (action, scoring target, threshold,point deduction, or the like) checked by the gymnastics judge duringscoring, and registers them in the scoring rule DB 16 for each athleticevent (S202).

Then, the preprocessing unit 21 determines the joint number related tothe skeleton model for each scoring rule ID (S203) and registers thecalculation formula for calculating the angle formed by a joint,received from the administrator or the like, in the calculation formulaDB 17 (3204).

Then, the preprocessing unit 21 receives, from administrator or thelike, the designation of the plane on which the circular arcrepresenting the angle of the joint is displayed and sets it in thecalculation formula DB 17 (S205). Then, in a case where the formedangles at multiple times are used for scoring, the preprocessing unit 21receives, from the administrator or the like, the designation of thenumber of times of measurements as needed and registers it in thecalculation formula DB 17 (S206).

[Flow of Comparison Display Process]

FIG. 15 is a flowchart that illustrates the flow of a comparison displayprocess. As illustrated in FIG. 15, the comparison display unit 23retrieves the video data on the performance of the player for arejudgement from the video data DB 15 and displays it (S301).Furthermore, the performance of the player for a rejudgement may bedesignated by the administrator or the like.

Then, the comparison display unit 23 extracts the joint position datarelated to the record ID of the retrieved video data from the jointposition data DB 14 (S302). Then, the comparison display unit 23 appliesthe extracted joint position data to a skeleton model to generate a 3Dmodel of the human body and displays it (S303). Then, the comparisondisplay unit 23 displays the 3D model in temporal synchronization withthe operation to reproduce the video data (S304).

[Flow of Process to Display Angle]

FIG. 16 is a flowchart that illustrates the flow of a process to displaythe angle formed by a joint. As illustrated in FIG. 16, the anglecalculating unit 24 acquires the event name with the record ID that isthe target to be determined from the video data DB 15 (S401) andacquires the scoring rule ID regarding the event from the scoring ruleDB 16 (S402).

Then, the angle calculating unit 24 retrieves the calculation formulafor each acquired scoring rule ID from the calculation formula DB 17 andcalculates the angle formed by a joint at the time when the performanceis reproduced (S403). Here, in a case where the number of times ofmeasurements is set to be more than one (S404: Yes), the anglecalculating unit 24 calculates the difference between the angles atmultiple measurement times in accordance with the calculation formula(S405).

Furthermore, in a case where the number of times of measurements is setto be not more than one (S404: No) or S405 is executed, the anglecalculating unit 24 displays the circular arc indicating the angle onthe plane horizontal to the reference plane by using the referencecoordinates for the calculation formula DB 17 as an origin in thedisplayed 3D model in conjunction with the selection button (S406).Furthermore, the angle calculating unit 24 changes the color of thecircular arc in accordance with a threshold (S407).

Advantageous Effect

As described above, as the video reproduction apparatus 10 calculatesthe joint angle of a player for a gymnastics judge in accordance with acalculation formula and displays it, it is possible to provideinformation for conducting scoring in the uniform manner in accordancewith a scoring rule. As a result, a gymnastics judge is capable ofscoring a performance in a detailed manner in accordance with a scoringrule defined for each event or element without depending on theindividual's habit, capability, experience, or the like.

Although the embodiment of the present invention is described above, thepresent invention may be implemented in various different embodimentsother than the above-described embodiment. Hence, a different embodimentis described below.

[Athletic Event]

Although the gymnastics such as pommel horse and still rings aredescribed as examples in the above embodiment, these are notlimitations, and other sports are also applicable, in which a playerperforms a sequence of elements and a judge gives a score. Examples ofthe other sports are other events of gymnastics, figure skating,rhythmic gymnastics, or cheerleading.

[Target Data]

Although the example using a video of the performance is describedabove, this is not a limitation. For example, still images may beprocessed in the same manner. Furthermore, a typical 3D modelingtechnology may be used as the technology for acquiring thethree-dimensional coordinates of each joint from the skeleton model.

[System]

The processing procedure, the control procedure, the specific name, andinformation including various types of data and parameters mentioned inthe above descriptions and drawings may be optionally changed if nototherwise specified. Furthermore, each component of each deviceillustrated is conceptual in terms of functionality and does not need tobe physically configured as illustrated in the drawings. Specifically,specific forms of separation and combination of each device are notlimited to those depicted in the drawings. A configuration may be suchthat all or some of them are functionally or physically separated orcombined in an arbitrary unit depending on various types of loads,usage, or the like. Furthermore, all or any of each processing functionperformed by each device may be implemented by a CPU and a programanalyzed and executed by the CPU or may be implemented as a wired logichardware.

[Hardware Configuration]

FIG. 17 is a diagram that illustrates an example of a hardwareconfiguration. As illustrated in FIG. 17, the video reproductionapparatus 10 includes a communication interface 10 a, an HDD (Hard DiskDrive) 10 b, a memory 10 c, and a processor 10 d.

The communication interface 10 a is a network interface card, or thelike, which controls communications with other devices. The HDD 10 b isan example of a storage device that stores a program or data.

Examples of the memory 10 c include a RAM (Random Access Memory) such asan SDRAM (Synchronous Dynamic Random Access Memory), a ROM (Read OnlyMemory), or a flash memory. Examples of the processor 10 d are a CPU(Central Processing Unit), a DSP (Digital Signal Processor), an FPGA(Field Programmable Gate Array), or a PLD (Programmable Logic Device).

Furthermore, the video reproduction apparatus 10 operates as aninformation processing apparatus that reads and executes a program toimplement a video reproduction method. Specifically, the videoreproduction apparatus 10 executes a program for performing the samefunctions as those of the preprocessing unit 21, the relating unit 22,the comparison display unit 23, and the angle calculating unit 24. As aresult, the video reproduction apparatus 10 may execute a process forperforming the same functions as those of the preprocessing unit 21, therelating unit 22, the comparison display unit 23, and the anglecalculating unit 24. Moreover, the program described in this differentembodiment is not exclusively performed by the video reproductionapparatus 10. For example, the present invention is also applicable to acase where the program is executed by a different computer or server ora case where the program is executed by them in cooperation.

The program may be distributed via a network such as the Internet.Furthermore, the program may be recorded in a recording medium readableby a computer, such as a hard disk, a flexible disk (FD), a CD-ROM, anMO (Magneto-Optical disk), or a DVD (Digital Versatile Disc) so that itmay be executed by being read from the recording medium by the computer.

According to the embodiments, it is possible to provide information forconducting scoring in the uniform manner in accordance with a scoringrule.

All examples and conditional language provided herein are intended forthe pedagogical purposes of aiding the reader in understanding theinvention and the concepts contributed by the inventor to further theart, and are not to be construed as limitations to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although one or more embodiments of thepresent invention have been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

What is claimed is:
 1. A non-transitory computer-readable recordingmedium having stored therein a performance display program that causes acomputer to execute a process, the process comprising: determining aposition of a joint of a performer based on a sensing result by asensor; calculating an angle formed by a joint used for scoring anelement of an athletic event by using calculation information regardinga calculation formula for calculating the angle and the determinedposition of the joint of the performer; and displaying a performanceimage of the performer acquired from a camera and a scoring image havingthe angle displayed on a 3D model image of the performer which isgenerated according to the sensing result so as to be compared with eachother.
 2. The non-transitory computer-readable recording mediumaccording to claim 1, wherein the determining includes, by referring topreviously measured and stored joint information for determining aposition of a joint of the performer, determining the position of thejoint of the performer on the performance image.
 3. The non-transitorycomputer-readable recording medium according to claim 1, wherein theprocess further comprises: receiving a designation of a specific timeduring a performance of the performer; and displaying the selectedperformance image and the scoring image so as to be compared with eachother by using the angle at the specific time.
 4. The non-transitorycomputer-readable recording medium according to claim 3, wherein theprocess further comprises: calculating angles at times previous to andnext to the specific time; and generating a new scoring image withregard to each of the previous and next angles.
 5. The non-transitorycomputer-readable recording medium according to claim 1, wherein theprocess further comprises: referring to a storage unit that stores thejoint used for scoring a corresponding element and the calculationinformation in relation to each of elements to determine the joint andthe calculation information corresponding to a selected element; andcalculating the angle used for scoring the selected element by using thedetermined calculation information.
 6. The non-transitorycomputer-readable recording medium according to claim 1, wherein theprocess further comprises: referring to scoring information that storesan angle used for scoring the element and a scoring criterion determinedin accordance with a difference from the angle in a related manner; andin accordance with the scoring criterion corresponding to the calculatedangle, changing a display color of the angle when the performance imageand the scoring image are displayed so as to be compared with eachother.
 7. A performance display method comprising: determining aposition of a joint of a performer based on a sensing result by asensor, using a processor; calculating an angle formed by a joint usedfor scoring an element of an athletic event by using calculationinformation regarding a calculation formula for calculating the angleand the determined position of the joint of the performer, using theprocessor; and displaying a performance image of the performer acquiredfrom a camera and a scoring image having the angle displayed on a 3Dmodel image of the performer which is generated according to the sensingresult so as to be compared with each other, using the processor.
 8. Theperformance display method according to claim 7, wherein the determiningincludes, by referring to previously measured and stored jointinformation for determining a position of a joint of the performer,determining the position of the joint of the performer on theperformance image.
 9. The performance display method according to claim7, further comprising: receiving a designation of a specific time duringa performance of the performer; and displaying the selected performanceimage and the scoring image so as to be compared with each other byusing the angle at the specific time.
 10. The performance display methodaccording to claim 9, further comprising: calculating angles at timesprevious to and next to the specific time; and generating a new scoringimage with regard to each of the previous and next angles.
 11. Theperformance display method according to claim 1, further comprising:referring to a storage unit that stores the joint used for scoring acorresponding element and the calculation information in relation toeach of elements to determine the joint and the calculation informationcorresponding to a selected element; and calculating the angle used forscoring the selected element by using the determined calculationinformation.
 12. The performance display method according to claim 1,further comprising: referring to scoring information that stores anangle used for scoring the element and a scoring criterion determined inaccordance with a difference from the angle in a related manner; and inaccordance with the scoring criterion corresponding to the calculatedangle, changing a display color of the angle when the performance imageand the scoring image are displayed so as to be compared with eachother.
 13. A performance display apparatus comprising: a memory; and aprocessor coupled to the memory and configured to: store calculationinformation regarding a calculation formula for calculating an angleformed by a joint used for scoring an element of an athletic event,determine a position of a joint of a performer based on a sensing resultby a sensor, calculate the angle by using the determined position of thejoint of the performer and the calculation information, and display theperformance image of the performer acquired from a camera and a scoringimage having the angle displayed on a 3D model image of the performerwhich is generated according to the sensing result so as to be comparedwith each other.
 14. The performance display apparatus according toclaim 13, wherein the processor is further configured to: by referringto previously measured and stored joint information for determining aposition of a joint of the performer, determine the position of thejoint of the performer on the performance image.
 15. The performancedisplay apparatus according to claim 1, wherein the processor is furtherconfigured to: receive a designation of a specific time during aperformance of the performer; and display the selected performance imageand the scoring image so as to be compared with each other by using theangle at the specific time.
 16. The performance display apparatusaccording to claim 15, wherein the processor is further configured to:calculate angles at times previous to and next to the specific time; andgenerate a new scoring image with regard to each of the previous andnext angles.
 17. The performance display apparatus according to claim 1,wherein the processor is further configured to: refer to a storage unitthat stores the joint used for scoring a corresponding element and thecalculation information in relation to each of elements to determine thejoint and the calculation information corresponding to a selectedelement; and calculate the angle used for scoring the selected elementby using the determined calculation information.
 18. The performancedisplay apparatus according to claim 13, wherein the processor isfurther configured to: refer to scoring information that stores an angleused for scoring the element and a scoring criterion determined inaccordance with a difference from the angle in a related manner; and inaccordance with the scoring criterion corresponding to the calculatedangle, change a display color of the angle when the performance imageand the scoring image are displayed so as to be compared with eachother.